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ORIENTATION OF PROGRAM 



Research in education, as in business and industry, must 
find its effectiveness in revised techniques, materials and pro- 
cesses at the local or production level. Technicians to utilize 
the research process and findings have been missing from most 
local school organizations while business and industry have bene- 
fited from the work of engineers, sales technologists and man- 
agement experts. On the local sconce in public education there 
is an emerging role which the American Educational Research 
Association is recognizing as a research utilization specialist at 
the county agent level. A training program for preparing per- 
sonnel to meet the role demands of such a position was conducted 
at the University of Kentucky during the period June 13, to August 
5, 1966. 

The participant group consisted of twenty-one males and 
three females ranging in age from twenty-three to fifty-five 
years. The median age was thirty-seven years. Nineteen of 
twenty-four possessed the master's degree and three others ex- 
pected to complete this degree with credit achieved in the insti- 
tute. Because the role for which training was offered is an 
emerging position in local school systems, participants in the 
institute came from several position titles. Principals and 
Assistant Principals made up the largest group ‘(15). Only one 
person entered the institute from the position of School Superin- 
tendent and only one came with the title Director of Federal Pro- 
grams. Four participants were classroom teachers who had been 
assigned additional duties in the preparation and/or evaluation of 
projects under Title I of Public Eaw 89-10. One participant was 
a supervisor of instruction, one a director of pupil personnel, and 
another was a guidance counselor. 

This institute program was predicated upon fourteen ob- 
jectives stated in behavioral terms. Role behaviors for research 
utilization specialist at the county agent level have been described 
in only very general terms i^i the literature of educational re- 
search. Therefore it was necessary to use a priori assignment 
of certain behaviors to arrive at the following objectives for this 
training program: 
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To be able to analyze a problem e ncountered in a 
local school system in sufficient and appropriate 
detail to provide researchable or developmental 
elements. 

2. To be able to designate the significant variables 
to be measured and controlled or predicted in 
the research design. 

3. To be able to locate and evaluate supporting or 
contributing past research findings in terms of 
their practical rather than theoretical contribu- 
tions . 

4. To be able to select a research design of no greater 
complexity than to involve analysis of variance, i. e. , 
randomized-groups design, randomized -blocks de- 
sign, factorial design. 

5. To be able to recognize and interpret ^'constant 
errors** and ^'random errors*' resulting from 
sampling and observational tecnniques, and to 
se lect populations and techniques which -will mini- 
mize these errors. 

6. To be able to select from among the available stand- 
ardized tests of achievement, intelligence, interest, 
aptitude, and objective personality instruments those 
most appropriate for a given school population and 
variable. Also, to be able to design a data-gathering 
device such as a questionnaire which is reliable and 
efficient. 

7. To be able to select and use statistical techniques of 
no greater difficulty than analysis of variance and 
product-moment correlation. Proficiency will not 
include ability to derive the theoretical origins of 
the mathematics involved in these methods. 

8. To be able to interpret statistical results into con- 
clusions which relate to variables easily identified 
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within the local school situation. 



To be able to explain in writing the method, data, 
conclusions and implications of research in a con- 
crete, practical and direct rather than abstract and 
theoretical m.anner. 



10. To be able to locate and interpret research findings 
which contribute directly (do not call for translation) 
to local school situation. 

11. To be able to derive the maximum applicable know- 
ledge from data routinely accumulated by the local 
school within the limits of his capability defined by 
levels of competency in items 4. , 5. , 6. , 7. , 8. , 
and 11., above. 

12. To be able to foresee and plan for the accumulation 
of useful information in the routine administrative 
and pupil inventory processes within the limits of 
competency defined in items 4. , 5. , 6. , 7. , 8. , 
and 11., above. 

13. To be sufficiently interested in research to make 
overt gestures directed toward influencing others 
to a concern for the scientific approach to educa- 
tional problem solving. 

14. To be able to utilize such techniques as ’’discussion 
groups,*' "panels,** "self-evaluations,** and *'role- 
playing** as well as traditional lecture-discussion 
techniques for the communication of research metho- 
dology and findings. 

These behavioral objectives are further crystalized in the 
goals for the preparation of participants: 

1. To conduct pilot testing of new practices and ma- 
terials . 

2. To make optimum use of existing data to interpret 
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the school program utilizing research methodology. 

3. To adapt and interpret available knowledge of educa- 
tional practices and practitioners. 
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DESCRIPTION OF THE PROGRAM 
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Educational research*methods and tools have been a 
traditional elenaent in graduate training for decades. Sut in 
most instances the content has been presented as courses rather 
clearly differentiated and too often apart from practical applica- 
tions. The institute format provided an opportunity to integrate 
the material into a whole which was designed to be comprehen- 
sible for students with only a meager preparation in the subject. 
As the program content is presented here the integrative aspects 
may not easily be discernable. Nevertheless, maximum effort 
was made by the staff to accomplish the inter- relatedness of con- 
tent and the interaction of tools. Numerous formal and informal 
staff conferences were held, staff members attended sessions 
together, team- teaching methods were employed, references 
were employed, references were made from one content area 
to another, and examples were used in successive sessions. 
Participants consistently commented upon the careful integra- 
tion of the content among instructors and consultants. There 
were no occasions of disagreement among staff members re- 
garding an element of content or method of presentation. 



For purposes of description and evaluation the program 
is viewed as having nine components. Brief descriptions of each 
of these elements are presented in the following paragraphs and 
detailed outlines are shown as Appendix A. 



1 . Statistics 

Content drawn from the area of statistics ranged from elementary 
description through the simplest techniques of statistical infer- 
ence. Participants could be described as having no preparation 
in statistics prior to this experience and it was necessary to move 
carefully and thoroughly from one topic to the next. Students were 
given individual help and small remedial groups were formed to 
assure everyone the opportunity for meaningful instruction. In 
the first two-week period two hours per day were given to lec- 
ture, discussion and demonstration relating to topics in statistics. 
Students solved selected problems and their work was corrected 
for immediate feedback to them. 
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2. Measurement 



Ten of the twenty-four participants had received credit for a 
graduate cours e in educational tests and measurement. Experi- 
ence revealed that this fact did not create the wide range of 

pr6p^r5lLlUXi WllXV^ll UlC S tcuxx 

sary to provide two levels of content and instruction. Instead 
the staff attempted to select from the large body of theory, 
techniques and devices in educational measurement a content 
containing the essentials for a research technician. Added to 
this foundation content was a description and brief encounter 



with several approaches to the quantification of observable be- 
havior. Topics which received concentrated treatment were: 
types and characteristics of scales: criteria for an acceptable 

measurement device or instrument; types of scores - their ap- 
plications and limitations; constructing items and tests; a- 
chievement measures; measures of intellectual function; ap- 
titude measures; personality and behvaioral measures; at- 
tempts to quantify teacher-behavior; the Q-sort technique; the 
Osgood sematic differential and profile analysis technique. 



3. Research Methods 



Content presented under this broad and non-specific title in- 
cluded an array of topics having as their single common quality 
a relationship with the' research process. Beginning with an 
identification of the components of the research process and con- 
tinuing with study of each element this area of content became 
the best opportunity for integration of learning in the program. 

Content can be very briefly described by the following statement 
of the research process: 

Identifying, delineating and describing a problem 
Developing hypotheses 

Collecting, evaluating, and synthesizing pertinent pre- 
vious research 

Designing a structure for the specific study 

Analyzing accumulated data 

Deriving conclusions and implications 
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4. Experimental Design 



While other research designs were covered under the topic of Re- 
search Methods, experimental designs .were given sufficient time 
and emphasis to be classified separately. After a detailed analy- 
sis of problems associated with the control of variables and con- 
ditions each of the popular experimental designs were examined. 
In group discussion students applied each of the designs to an edu- 
cational research situation noting the adequacy of the particular 
design for controlling circumstances likely to be encountered. 
Statistical treatments within the repertory of the students were 
applied to the designs for which they were suitable to illustrate 
uses and associate meanings. 

5. Evaluation of Research 

Certain subject matter and activities in the program were di- - 
rected to the improvement of students' abilities to evaluate the 
research of others. Knowledge of appropriate procedures and 
understanding of meanings within the content relating to the re- 
search process conveyed to students a context for evaluating re- 
search reports. A set of guidelines was examined and applied 
in the evaluation of research reports selected by each student. 

6. Statistics Laboratory 

One hour on alternate days during the first six weeks of the in- 
stitute was devoted to learning the operation of desk calculators 
and to the development of speed and accuracy with them. Prob- 
lems selected for practice were related to educational research. 
Raw data were treated to describe populations and to test hypo- 
theses about treatment effects. Students completed several exer- 
cises drawn from probability theory using dice, poker chips with 
recorded test scores on each side and normally distributed color 
frequencies, playing cards, a table of random numbers, and 
other devices. 

7. Research Writing 

Utilizing the content of selected topics of the program students 
produced written materials for criticism and feedback. Each 
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student prepared a written statement of the problem component 
of a study. To assure equal difficulty and to provide a common 
base for criticism a research problem was identified by the group 
and analyzed individually for reporting. Products of this experi- 
ence were evaluated on the following criteria: 

Adequacy of handling of each component of the problem 

Vocabulary usage and definition of terms 

Handling of grammar and punctuation 

Handling of references 

Students developed hypotheses relating to the specific problem. 
These hypotheses were written on the blackboard, discussed by 
the students who develoned them, and criticised by the group. 

8. Field Trips 

After hearing a one-hour lecture on the characteristics and uses 
of computers in educational research the participants were given 
a one-hour tour of the University Computer Center. Content of 
the lecture included: an introduction to binary arithmatic with 
applications to machine technology; components of a computing 
system and their interaction; entering and controlling the com- 
puter; brief introduction to systems logic. The tour provided 
an opportunity to witness the complete processing of data from 
raw form to print-out with brief descriptions of each step in the 
operation. 

A second field trip consumed one-half day. The group traveled 
by bus to Spindletop Research, Inc. located five miles northwest 
of Lexington, Kentucky. Spindletop Research is a private organi- 
zation with a professional staff of 64 and a technical and office 
support staff of 44 employees. Students were given a half-hour 
briefing which introduced them to the organization and to the prob- 
lems peculiar to seeking funds by preparing research proposals. 

A guided tour of their facilities provided an opportunity to observe 
research being done by metalurgists, chemists, bio- physicists, 
communications experts, sociologists, and psychologists. An 
interdisciplinary approach is used to conduct research in com- 
munications, comparative effectiveness, economic development, 
environmental sciences, and industrial sciences. 
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9. Consultanto 

Content presented by the visiting researchers tended to be gen- 
eral and non-task oriented. Interaction between the visitors and 
participants was evident very soon after initial contact lowered 
the barriers to communication. Techniques involving inductive 
teaching were effectively demonstrated by each of the consultants. 
Perhaps their most significant contribution to the institute experi- 
ence was to provide an occasion for students to express ideas 
employing language, concepts and ideas drawn from the content 
of research methodology, Unfortunately their visits came too 
early in the program for these opportunities to be of maximum 
benefit. 
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days were given to statistics laboratory and small group discussions. 



An institute format provides for the flexible schedule 
necessary for completing irregular blocks of content and for 
integrating learnings across subject matter lines. However, 
this flexibilitv comnlicates the d 

^ 4 ^ Vfx U440 XXAOtXLUttS 

schedule of instruction. While the time intervals suggested 
for each topic are reasonably accurate they were not followed 

when student interests or needs required more or less than these 
time allocations. 

Irregular events such as visits by consultants and field 
trips were fitted into the time schedule as necessary. Consul- 
tants attended each of the instructional periods during their 
visits and they were actively involved with the group. Their ap- 
proaches ranged from assuming full responsibility for the con- 
tent of the program to serving as a specialist in discussion groups 
Break-times and study periods provided each participant the op- 
portunity to interact with consultants and staff members on a per- 
sonal and informal basis. Field trips were taken during periods 

of times which did not subtract from the usual instruction and 
discussion. 



EVALUATION OF THE PROGRAM 



F rom the vantage point of today many of the difficulties 
encountered in bringing this institute from an idea to a reality 
seem to be lost in the wholeness of the experience. While the 
character of the institute was abundantly influenced by solutions 
to problems each successive event lost a portion of its initial 
impact and significance. A few problems occasioned by the 
chain of events leading to late approval of the proposal, could 
not be overcome and their influence was constantly evident as 
the institute experience unfolded. It is in the attempt to evalu- 
ate that this influence becomes problematic. Details, particu- 
larly troublesome ones, often receive unjustified attention when 
it is the configuration which should be evaluated. These trouble- 
some details are cited in retrospect and with reluctance while 
the totality of the institute’s impact upon participants and staff 
members is emphasized. 

On December 6, 1965 the proposal for a Special Institute 
for Training Research Utilization Specialist for Local School 
Systems was transmitted from the University of Kentucky Re- 
search Foundation. For a period of two months this proposal 
lay unnoticed in the Receiving Department of the Bureau of Re- 
search, Office of Education. On April 21, 1966 notice of ap- 
proval of the proposal was received in the office of the Univer- 
sity of Kentucky Research Foundation. Then followed a period 
of more than six weeks before signing of the contract on June 
16, 1966 . The following conditions were predisposed by this se- 
quence of events: 

1. Dr. Ernest McDaniel who was to have been a staff member ac- 
cepted other employment for the summer session. It became 
a serious problem to find a replacement for him. 

2. A secretary to prepare materials and assist in recruiting par- 
ticipants could not be employed until May 26, 1966. Other 
secretarial assistance was scarce and unenthusiastic. 

3. It was not possible to secure consultants at appropriate times 
during the institute period because they had scheduled their 
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time for other activities. 



4. A heavy summer schedule had consumed all of the desirable 
space at the University of Kentucky making it necessary for 
institute participants to move from room to room often find- 
ing another group using the assigned quarters. 

5. Requirements for admission to the graduate school of the 
University of Kentucky made it impossible for late applicants 
to receive admission except on a special temporary basis. 

6. It was not possible to give the institute publicity through news 
media because of a directive contained in the letter of notifi- 
cation of approval. "Please do not release information to the 
public concerning the possible Federal support of the project 
until the contract has been signed by your contracting officer 
and the contracting officer for the Office of Education. " 

7. There was not sufficient time to publicize the institute, re- 
ceive applications, screen applications, notify applicants who 
were approved, and select replacements for applicants who 
declined appointments. 

» 

Staff members and participants attempted without success 
to evaluate the influence of these several problems on outcomes 
of the institute. No doubt, at the time of occurrence each problem 
was significant, but human qualities have a way of absorbing in- 
conveniences and functioning to compensate for imbalances which 
might result. There is reason to believe that the particular set 
of circumstances which preceeded the institute and continued to 
emerge in the form of new problems served to create a strong 
force for success. Among the conditions which lead to this ob- 
servation are these: 

1. A group feeling of closeness and rapport developed 
very early among participants. This is a usual pattern in the 
institute approach to training, but in this group it developed more 
rapidly than staff members would have predicted. 

2. Staff members openly shared the difficulties with 
the trainee group and as a result became more involved with 
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student problems and needs tb:m is ordinary. 

3 The amount of work accomplished by staff members 
and participants was extraordinary which indicates a personal 
committment on the part of each member to contribute to a suc- 
cessful experience. 

4. Staff members sought every opportunity to be of 
assistance to each other. Neither the close of a session, the 
end of a day, nor the termination of the program brought these 

offers to an end. 

It is sufficient to report that circumstances which created 
hardship concurrently stimulated human resources toward suc- 
cessful solutions. 

Cooperation and assistance certainly was not limited to 
the immediate staff and to the trainee group. Persons in the 
several offices of the University of Kentucky were ready to lend 
their assistance within the limitations of reality. The University 
of Kentucky Research Foundation arranged to provide postage for 
the mailing of announcements and applications when it was not 
possible to recover this money through the grant because of the 
long delay in completing the contract and the establishment of an 
effective date for reimbursement of funds. Secretarial time was 
given by several departmental secretaries. Room assignments 
were rearranged by the very cooperative summer school staff. 

The College of Commerce shared its calculators and provided 
keys to their statistics laboratories. Office space was provided 
by the Bureau of School Services in their Cooperative Testing 
Services quarters. The institute director and staff are indebted 
to the fine people who made these arrangements at the last minute 
and under conditions which would have made it easy and reason- 
able for them to have refused. 

Evaluation of the Objective Component 

Fourteen objectives phrased as achievement goals for 
trainees were the purposes for which this program was offered. 
There are no data for evaluating the validity of these objectives 
as necessary levels of performance for local utilizers of research 
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tools and information. But it is possible to estimate the level 
of accomplishment of each objective attributable to the program 
of the institute. Trainees completed a questionnaire on the final 
day of the program. One element called for an estimate of the 
level of group accomplishment of each objective. Results of 
this item appear in the following table: 
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Table 1 



Profile of Mean Ratings 

Assi<^Tied to Accomolishment Levels of Institute Obiectives 

O A" 




Student ratings indicate that the objectives were more 
than moderately achieved by the institute experience. Highest 
levels of achievement were noted in: 

13. To be sufficiently interested in research to make 
overt gestures directed toward influencing others 
to a concern for the scientific approach to edu ca- 
tional problem solving. 

7. To be able to select and use statistical techniques 
of no greater difficulty than analysis of variance 
and product-moment correlation. 

Lowest levels of accomplishment were assigned to the objectives: 

14. To be able to utilize such techniques as •’discussion 
groups”, ’’panels'', ’’self- evaluation”, and "role- 
playing” as well as traditional "lecture-discussion” 
techniques for the communication of research metho- 
dology and findings. 

1. To be able to analyze a problem encountered in a 

local school system in sufficient and appropriate de- 
tail to provide researchable or developmental ele- 
ments. 

Analysis and evaluation of the program of the institute will contri- 
bute to an understanding of possible reasons for the ratings of cer- 
tain objectives to be high and others somewhat lower. 

Staff members agree with students in every aspect of their 
estimate. Faculty opinion is well summarized by Dr. James Eaves’ 
evaluation comment. "Student growth was actually more than one 
could demand though still short of our r :;pirations for them." Uiz 
evaluation of the accomplishment of g. als having statistical con- 
tent is reported in the following paragraph: 

« 

"Group members were mature and, in a way, almost homo- 
geneous in their previous work in statistics. While many had a 
meager knowledge of descriptive aspects of the subject, none had 
gained a usable knowledge of statistics. A few had an acceptable 
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mathematics preparation, but most were sorely lacking in this 
discipline. Not one spoke the language and none were willing to 
talk about a population in relevant statistical terminology. They 
preferred to add the scores up or, better still, to watch some- 
one do this, get an answer, and quit. They were all horrified at 
the thought of interpreting results. The presentation of any new 
topic brought on more anguish, followed by confusion, then study 
and determination, followed by acceptance; and all of this suc- 
ceeded by a feeling of acquaintanceship, of usefulness, of applica- 
tion, understanding, and often a surprising degree of mastery. 

At first their response was slow and accompanied by a rebellious 
attitude. Their surrencer to scholarly study brought purpose 
and direction to their fitful struggles and movement became evi- 
dent. Once the students gained understanding and confidence 
they moved swiftly, with comprehension, toward mastery of the 
concepts. Even now these participants are not competent re- 
search statisticians but they have been subjected to a great deal 
of material pertinent to research requiring statistical methods 
and they know their limitations as well as their competencies. 

The overall development of this group has been phenomenal. 

They possess much more of a professional aire as well as a 
mature respect for their own research prospects. The school 
systems to which they return will profit from their determined 
efforts in a difficult area of study. ” 

Dr. John Anglin, in his evaluation statement said, ‘'IJhe 
participants were determined in their efforts and diligent in their 
quest for understanding of the role of measurement in the re- 
g0a.j*ch activites of local schools. Although the training and ex- 
perience of individuals was varied, the flexible nature of the 
institute program permitted a notable growth for each member. 

By performance and expression participants have exhibited a 
positive attitude toward research and a significant gain in their 
understanding of the research process." 

Evaluation of the Program by Participants 

Realizing that instructors may possess superior know- 
ledge of research methodology and experience still it is neces- 
sary to turn to the evaluations by students to assess the imme- 
diate impact of the institute program. An aspect of the evaluation 



questionnaire requested students to make judgments concerning 
the program. First they reacted to an item relating to the amount 
of time given to each of the nine elements of program described 
in preceeding pages. The item and directions of responding are 
contained in the questionnaire, Appendix B. Results of partici- 
pants’ judgments appear in the following table: 

Table 2 



Frequencies of Participants' Judgments Concerning Appropriate 
Time Allocations For Subjects and Activities 



Topic 

or 

Activity 


Time Allottment Should be 


Increased 


Retained 


Decreased 


Statistics 


1 


11 


12 


Measurement 


4 


20 




Research Methods 


7 


17 




Experimental Design 


13 


11 




Research Evaluation 


13 


11 




Research Writing 


19 


5 




Statistics Lab 


2 


17 


5 


Field Trips 


4 


19 


1 


Consultants 


4 


13 


7 



Table 2 contains information very important to this evalua- 
tion. Disagreement regarding the appropriateness of time alloca- 
tions is apparent. Examination of the suggested percents of time 



increase or decrease provides more information and analysis of 
student response interactions between program, elements reveals 
substantive differences. These findings are discussed in the 
following paragraphs. 

The group was about equally divided in their judgments con- 
•'erning statistics. Perhaps most significant is the suggestion by 
ne-half the group that the time allotted to this topic be decreased. 
Of these twelve who indicated this opinion, four also suggested a 
decreas* in statistics laboratory time. In the first two-week 
period wnen statistics consumed two hours per day and the labo- 
ratory another hour anxieties and hostilities were quite evident. 
Staff members have estimated that one -half of the group were 
severely disturbed by frustrations resulting from failure to com- 
prehend matters in statistics. Questionnaires were kept anony- 
mous and it is not possible to know which participants made a 
particular suggestion; however, it is reasonable to estimate 
that the anxious students suggested less time for statistics. The 
laboratory provided an opportunity for students to gain assis- 
tance in problem solving and further instruction which may ac- 
count for the fact that eight students suggesting a decrease in 
statistics suggested that laboratory time was appropriate. How- 
ever, suggestions that laboratory time should be increased did 
not come from this group. Because of the heavy concentration 
of time for statistics in the early phase of the program it cannot 
now be determined whether this factor or a general desire to de- 
emphasize the topic caused this reaction among the students. 
Another element possibly active in the outcome was the instruc- 
tor's concern that students understand statistical concepts as 
■'^'ell as be able to solve problems. A look at the ratings given 
to goal achievement by the group suggesting a decrease in statis- 
tics reveals an interesting relationship. While making this sug- 
gestion they rate the objective pertaining to the appropriate se- 
lection and application of statistical techniques, as having been 
realized at a high level (4.4). An interpretation which seems to 
fit this outcome and the observations of the staff is that although 
the experience was painful it contributed to the successful attain- 
ment of an objective which might not have been reached with less 
time and emphasis. 

Students were in general agreement that time allotted for 



measurement was appropriate but it was suggested by four per- 
sons that the time should be increased by 25 percent. Three of 
these four were in the group suggesting a decrease in time de- 
voted to statistics. 

A sizeable group (7) suggested an increase of 25 to 75 
percent in the time given to the topic identified as research 
methods. Only two of these seven persons came from the group 
suggesting a decrease in statistics. One interpretation of the 
suggestion to increase time allotted to this rather general con- 
tent is that it related to problem identification and analysis which 
was rated as less fully accomplished than most of the other ob- 
jectives of the program. A weakness of the group as they were 
assessed by staff members and consultants was evidenced in their 
inability to conceptualize educational problems in terms of possi- 
ble causes and effects. The staff generally agreed that this weak- 
ness sprung from a lack of preparation in the sciences of human 
behavior. Another weakness of the participants as a group ap- 
peared in their handling of logic. It was often necessary to give 
extended time to drawing out relationships among variables and 
the deduction of possible outcomes of given actions. Based upon 
this interaction of the rating of goal accomplishments with judg- 
ments of time allottments suggestions for more time seem 
reasonable. 

Experimental design as a topic brought forth a division 
of opinion between increasing its time allottment by 25 percent 
and leaving it constant. Since over half of the group (13) sug- 
gested an increase further analysis seems to be indicated. This 
group assigned a value of 3.4 to the level of accomplishment of 
the objective which relates to the ability to select a research de- 
sign; whereas, the group indicating that the time was adequate 
rated this objective at a 4. 0 level. It is reasonable, with this 
relationship established, to assume that those who suggested 
more time felt that such an increase might bring closer the a- 
chievement of this goal. Since the same instructor presented 
the content of research methods and experimental design it is 
important to note the strong interaction between these two topics. 
Six of the seven who suggested increased time for research 
methods came from the group suggesting increased time for ex- 
perimental design. Looking further it is evident that the two 
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additional topics presented by this instructor, research evalua- 
tion and research writing, were given less time than most of the 
participants felt they needed. Since the combined content of 
these four areas was given only one hour per day the judgments 
cited here are likely to reflect justifiable criticism of the time 
allottment. Additional evidence supporting this interpretation 
is cited in the analysis of observed relationships between ac- 
complishment of objectives and each program element. 

It is important that seven participants felt that the time 
given to consultants should have been decreased by 25 to 50 per- 
cent. Since the amount of time for consultants and the schedules 
of their visits were determined by their availability this obser- 
vation may reflect upon the preparation of the participants to 
receive and interact with the consultants. General comments 
made by the participants, group behavior, and observations of 
staff members all lead to the evaluation that each consultant 
contributed significantly and favorably to the institute program. 

Students were generally pleased with the time given to 
field trips although a small number felt thex’e was not enough 
time. Data are not available to determine whether these four 
would suggest more trips or more time for the trips which were 
taken. 



Another element of the questionnaire requested that a 
rating ranging from 1 (little or no contribution to the objective) 
to 5 (the major contributor to the objective) oe assigned to each 
of the fourteen objectives. Table 3 shows the mean rating as- 
signed to each program component and each objective. 








Table 3 



Participants’ Mean Ratings Assigned Each Program Component 



Program 


Objectives 


Components 


1 


2 


3 


4 


5 


6 


7 


8 


9 


10 


11 


12 


13 


14 


Statistics 


2.9 


2.4 


2. 1 


2.9 


4. 2 


2.4 


3. 7 


3. 3 


2. 4 


1.9 


3.5 


2. 6 


3. 2 


1.8 


Measure- 

ment 


3. 0 


3. 0 


2. 7 


2.9 


3. 3 


4. 6 


3. 7 


3. 2 


3. 2 


o 

CO 


2.8 


4.0 


3. 5 


3. 7 


Research 

Methods 


4. 0 


3. 9 


4. 1 


3.7 


3. 5 


3.4 


3. 5 


3. 7 


4. 3 


3. 8 


3.6 


4. 0 


4.4 


3. 6 


Experimen- 
tal Design 


3.4 


4. 0 


3.5 


4. 1 


3. 2 


3. 1 


2.9 


3. 3 


3. 6 


3. 3 


3.3 


3. 6 


4. 0 


3. 1 


Research 

Evaluation 


2. 9 


2. 9 


3.3 


2.9 


2. 6 


2. 9 


2.4 


3. 0 


3. 1 


3.9 


3.2 


3. 0 


3. 5 


2. 7 


Research 

Writing 


2. 3 


2. 5 


2. 8 


2.4 


2. 1 


1.8 


1.8 


2. 5 


2. 5 


2. 8 


2. 3 


2. 1 


2. 8 


2. 0 


Statistics 

Lab 


1. 5 


1. 3 


1.3 


1.3 


1. 6 


1.2 


2. 6 


1.5 


1.2 


1.5 


1.4 


1. 3 


1.4 


1.2 


Field 

Trips 


1. 5 


1. 3 


1.2 


1. 1 


1.2 


1.2 


1.2 


1.2 


1. 3 


1.3 


1.3 


1.5 


1.8 


1.5 


Consul- 

tants 


3. 0 


3. 0 


3. 0 


2. 7 


2. 5 


2. 3 


2. 1 


2.4 


2. 4 


2. 1 


2. 5 


2. 7 


3.4 


3. 0 






1 
















J 











Without further analysis the content of Table 3 leads to 
only one general comment. Students saw each topic and activity 
as contributing in some amount to each objective. This observa- 
tion strengthens the judgment that the institute program was inte- 
grated across subject matter lines. But for a more thorough under- 
standing of the meanings conveyed by these data it is necessary to 
synthesize student judgments with original program plans and with 
other phenomena. 



At the outset each program component was seen as relating 
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directly to a certain constellation of objectives. While student 
opinions generally support the initial assignment of .responsibility 
to content areas a few differences are evident. Most noticeable 
among these deviations is the judgment associated with reaching 
the objective ”to be able to select and use statistical techniques. ** 
Statistics class sessions and laboratory activities were expected 
to produce the most significant advances toward this objective. 
However, the ratings assigned to these and other program ele- 
ments for Objective 7 must lead to another conclusion. The com- 
ponents listed as measurement and research methods were rated 
as contributing almost as much as the statistics component to 
the attainment of this objective. It may be that the most reason- 
able interpretation of results of student opinions relating to the 
topic of statistics derives from the phrasing of the objective and 
the nature of instruction in other topics. Perhaps key words in 
the statement of the objective are ’’select** and **use**. Content 
of the statistics sessions focused upon the learning of somewhat 
discrete elements and the comprehension of relationships among 
these elements. In the content of measurement and research 
methods students encountered realistic applications of these sta- 
tistics. It is probable that in these opportunities to observe ap- 
plications students gained insight which shed meaning to their 
previous knowledge of facts. 

Student views of the contribution of the statistics labora- 
tory to learnings associated with this objective are dis sappointing. 
Although problems constantly arose in terms of facilities, machine 
malfunctions, too few calculators, and equipment which did not 
arrive because of a late ordering date the laboratory experience 
was expected to contribute more significantly to the program. It 
is possible that their ratings reflect the hostile feelings of stu- 
dents rather than their true estimate of the value of this labora- 
tory experience. 

Among all the ratings given to subject areas the highest 
was 4. 6 assigned to measurement in reaching Objective 6 which 
was *'to be able to select from among the available standardized 

tests those most appropriate for a given school population 

and variable. ’* Students indicated on a previous item their judg- 
ment that this objective was accomplished at a 4. 0 level on a five 
point scale. Four other objectives were accomplished at an equally 



high or higher level. Therefore, the meaning of the 4. 6 rating 
must be associated with the specific contribution of the sessions 
on measurement to the ultimate level of accomplishment of the 
objective. In light of the generally high ratings assigned to 
measurement for all objectives the 4. 6 might have been inter- 
preted as having resulted from a personality variable. 

The topic "Research Methods" was initially expected to 
be related to several institute objectives. An examination of the 
ratings assigned to this topic for each objective reveals more 
than a moderate contribution in every instance. Its highest 
ratings occurred on objectives 13., 9., 3., 1. and 12. in that 
order. Objective 13. related to the development of interest in 
research which students rated as the goal most nearly achieved. 
Objective 9. related to written expression of research ideas and 
processes; Objective 3. deals with the location of past research 
findings; Objective 1. encompassed the ability to analyze a prob- 
lem; and Objective 12. sought to develop the ability to foresee and 
plan for accumulation of data. Since these objectives form the ar- 
ray about which the major content of this element of the program 
centered it is assumed that the topic served to accomplish its in- 
tended purpose. 

Research evaluation and research writing had their antici- 
pated levels of contribution to goal achievement except in one im- 
portant instance. A mean rating of 2. 5 given to the topic of re- 
searclp. writing in the accomplishment of the goal related to that 
topic is surprising. By student opinion the topic contributed 
more to several somewhat alien objectives than to its most 
closely associated goal. Meanwhile three other topics contri- 
buted significantly to the writing objective. The only imme- 
diate explanation of this situation is that the topic and activity 
provided for students was neither well planned nor well executed. 

Since consultants to institutes rarely direct their efforts 
to the achievement of specific goals it is not surprising that gen- 
erally moderate to low ratings were assigned them. Probably 
this condition would have been less evident if the institute direc- 
tor had planned the consultant contribution more carefully. An 
explanation may be found in the fact that the director did not feel 
that in an institute of this kind a consultant could provide a truly 
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valuable assistance. It is possible that this feeling was com- 
municated to the students and reflected in their responses. 



Trainees were asked to 
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were used by staff members, 
resulting from these items. 



respond to items of the questi on- 
to the instructional methods which 
TS’ble 4 contains the frequencies 



Table 4 



Frequencies of Responses on Items Relating to Instructional 

Methods 





Categories 


Items 


Too 

Frequent 


1 About 
Right 


Too 

Infrequent 


Use of lecture 


7 


17 




Class discussion 




19 


5 


Instructional aids 




9 


15 


Illustrations, examples 




18 


6 


Opportunities to question 




24 





The troublesome problem of maintaining balance between 
lecture and class discussion is evidenced by these responses, 
ust less than one-third felt lecture was too frequent and almost 
one- fourth indicated a need for more class discussion. Since 
three instructors were involved it is difficult to assign a mean- 
ing to these responses. The lecture -discussion ratio varied 
rom one instructor to another and from one topic to another. 
Interpretation must be limited to an acknowledgement that the 
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institute failed to satisfy the stated desires of a significant por- 
tion of the group in the matter of opportunities to discuss topics. 

At first glance the responses to use of instructional aids 
represents a strong criticism of the teaching staff. In part this 
nnay be true but two factors beyond the control of anyone could 
have produced the feeling reflected in the response. Laboratory 
equipment for use in the study of probability was not available 
for student use because of delays in ordering. Calculators were 
few in number and many of them were in need of maintenance. It 
is true that films were used only once in the institute, but dupli- 
cated material was a frequent part of the instructional plan. This 
may be a justifiable criticism of the program and certainly with 
more time for pre- institute planning the use of instructional aids 
can be improved. 

Trainees’ estimates of their own personal development 
during the institute is of importance in evaluating the experience. 
One element of the questionnaire sought to assess these personal 
changes according to the self-report of participants. Table 5 
shows the mean gains reported on each variable using a five 
point scale where a value of one is low and five is high. 
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Table 5 



Indicated Mean Gains of Participants on Six Categories 



Categories 


Initial Mean 


Final Mean 


Gain 




Value 


Value 




Knowledge of subject matter 


1.8 


3. 8 


2.0 


Attitude toward research in 








educ ation 


3. 0 


4. 5 


1.5 


Ability to conceptualize prob- 
lems 


2.4 


3.8 


1.4 


Ability to evaluate research 


2. 1 


3. 8 


1. 7 


Ability to design and carry 
out research 


1.5 


3. 7 


2.2 


Ability to communicate re- 








search matters to others 


1. 7 


3. 8 


2. 1 



Significant personal gains were indicated on each of the 
six variables. While attitude toward research in education 
reached the highest mean level it also began at the highest level. 
Largest gains were noted on research ability, communication of 
research matters, and knowledge of the subject matter of the 
institute. Lowest mean gain was observed in the ability to con- 
- ceptualize problems in education. This result is in support of 
comments made earlier in reference to the trainees' handling of 
the problem component of a research effort. 

Reactions to other items involving personal characteristics 
provide important information. Eleven of the twenty-four indicated 
that their degree of personal involvement was more than ever in a 
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course in the past. Seven were involved at their typical level 
while five indicated less involvement than ordinary. Three were 
distracted from institute matters by other concerns; an automo- 
bile accident with prolonged minor injury; an immient surgical 
appointment; and an illness. Four participants indicated some 
lack of interest because the material was not suited to their per- 
sonal needs. The remaining participants did not respond to the 
item. 

Staff and Teaching Assignments 

Without question the single most important determiner of 
learning in an institute as well as any other teaching-learning 
situation is the quality of the teaching staff. This institute, while 
not blessed with a staff having a significant national reputation 
in educational research, provided an instructional experience of 
quality for this unique student group. Participants were free 
with their discussion of instructional characteristics of indivi- 
dual staff members and naturally these comments were passed 
along by students, professors and administrators not associated 
with the institute. During the institute period and immediately 
thereafter accounts of student comment flowed constantly to the 
ear of the institute director. Not one such comment was less 
than complimentary. Without solicitation letters have come 
from institute participants since the closing day expressing high 
regard for the instruction which they received this summer. 

Qualities which contributed to an optimum instructional 
experience were patient understanding, knowledge of subject 
matter, verbal facility, cooperative spirit, unsellfish sharing 
of time, enthusiasm shown in contact with students, and a strong 
desire to provide a learning experience suitable to individual stu- 
dent levels of readiness and need. 

Staff members and their teaching assignments: 

Ivan L. Russell, Ph. D. , Institute Director 

Presented topics of research methods, experimental design, 

evaluation of research, and research writing. 

James Eaves, Ph. D. , Professor of Mathematics 
Presented topics in statistics and conducted the laboratory 
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John Anglin, Ed. D. , Assistant Professor of Education 
Presented topics in measurement 

Mr. Robert Brown, M. A. in Mathematics and Computer 
Science 

Statistics and Laboratory Assistant 

Malcolm Provus, Ph. D. , Director of Research 
Pittsburgh City Schools 
Pittsburgh, Pennsylvania 

Full-time with the institute during the period July 20 to August 
2. Assisted in all elements of program presented during that 
period. 



Consultants to the institute: 

Dr. J. Thomas Hastings, Director 

Center for Instructional Research and Curriculum Evaluation 
University of Illinois 
Urbana, Illinois 

Dr. Robert M. Rippey, Associate Director 
Center for the Cooperative Study of Instruction 
University of Chicago 
Chicago, Illinois 

The number of institute staff members seems reasonable 
when only student- teacher ratio (approximately 10:1) is considered. 
However, when the array of topics covered and the daily class 
time are taken into account it becomes evident that additional 
teaching staff were needed. With the initial approval of the insti- 
tute proposal there was a strong suggestion that a director of re- 
search from a large urban school system be employed as a faculty 
member. A thorough search for such a person available for the 
entire eight week period was futile. The best arrangement which 
could be substituted for this suggestion was to employ Dr. Provus 
of the Pittsburgh Public Schools for a short period of time. In 
summary, while the staff was marginally adequate in terms of 
number their enthusiasm and unselfish effort produced an ef- 
fective instructional situation. 
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Selection and Characteristics of Trainees 



On April 20, 1966 a total of 940 notices of the institute 
were mailed to schools in Kentucky, Indiana, Ohio, Illinois, 
Missouri, Tennessee, Virginia, and West Virginia. Copies 
were sent to the Superintendent of Public Instruction in each 
of these states. The announcement and the letter of transmittal 
are shown in Appendix C. As a result of this publicity 124 in- 
quiries were received and applications mailed during the period 
April 23 to May 20, 1966. Of these applications 52 were returned 
in completed form. The following criteria were applied to select 
the trainees: 

1. A baccalaureate degree and two years of teaching or 
administrative exper'Cxice. 

2. Responsibility in the research program for a local 
school system. 

3. If graduate credit was sought it was necessary to 
be eligible for graduate school admission which requires: 

a. Graduation frcxxO. an accredited college 

b. An overall undergraduate grade point average 
of 2.5 on a 4. 0 scale. 

Each of the candidates finally selected returned letter or 
telegrams indicating their intent to attend the institute. However, 
eight failed to appear on June 13 for the beginning of the program. 
Only two of the eight returned correspondence indicating their 
inability to attend. These two were replaced by lesser qualified 
candidates but there was not time to recruit and select replace- 
ments for the other six positions. 

The participant group was representative of a wide geo- 
graphic region in the State of Kentucky, One trainee came from 
southwestern Illinois (Belleville). School systems large, medium 
and small were represented. While diversity of location and back- 
ground was characteristic of the group they brought with them 
many communalities . They were capable in the basic and daily 
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routines of public education, experienced in the problems of 
teachers and administrators, and directed by a desire to seek 
new answers to old problems. Although a tendency to think in 
the stereotyped cliches was evident they quickly moved toward 
more open consideration of matters. Perhaps the best descrip- 
tion would mention their earnestness of purpose, willingness to 
work, and tolerance for frustration. In brief, the problems of 
this institute cannot be associated with the characteristics of its 
participants. Generally, the group was able to meet the demands 
of the program and to develop a remarkable degree of capability 
in educational research as a result of the program and their in- 
dividual efforts. 

Organization 

Daily schedules, classroom facilities and other organiza- 
tional matters have been discussed in detail under other topics. 
But an evaluation of the length of the program remains of some 
concern. Unquestionably the period of eight weeks allowed time 
for only an introduction to the problems and methods of educational 
research. And there is no question that the accomplishments of 
this institute were limited to introductory levels of personal 
achievement. Perhaps the real question is ’’Can persons with 
a minimum preparation be g^ven a worthwhile program in an 
institute of eight weeks? ” 

While an answer based upon objective data must await the 
outcomes of personal efforts by the institute participants a sub- 
jective opinion can be stated. Despite a very ambitious effort to 
present a wide array of topics the staff has expressed the opinion 
that the experience was worthwhile and should be repeated with 
refinements. Certainly the short institute is a better setting for 
research training than the traditional three or four scattered 
courses with relatively unrelated content. 
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EVALUATION SUMMARY AND CONCLUSIONS 
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this institute program there are several details and circum- 
stances -which were troublesome and require improvements. 
Highest in the order of priority for change is the content organi- 
zation of the program. Assuming a trainee group with similar 
characteristics a future program should contain these elements: 



1. Much more time should be given to conceptualizing 
research problems and analyzing them. It is not reasonable to 
assume that trainees will be able immediately to select the signi- 
ficant elements of a problem situation and to view these elements 
as variables in a research endeavor. Becoming analytic in ap- 
proaching a problem is not easy for persons who have not been 
prepared to view problems in this manner. Undergraduate educa- 
tion, indeed general public education, has not encouraged and 
stimulated the learner toward inquiry. Years of experience in 
passive learning and careless acceptance of ideas do not prepare 
the individual for the kind of thought processes required of a re- 
searcher. Despite some careful attention to the problem com- 
ponent this institute was not successful in preparing these trainees 
in problem development. 



2. It would be wise to withhold the introduction to statis- 
tics for the first two weeks and to devote this time to concept 
building. Probably in the first two weeks a group could be pre- 
pared to receive instruction in statistics by helping them to see 
the need for certain data treatments. 



3. More time should be given to discussing research 
methods and experimental design. Trainees should have frequent 
experiences in the selection of experimental designs for selected 
problems in education. Opportunities to evaluate the suitability of 
a design for a particular study would provide the familiarity and 
insight necessary for learning. 

4. One consultant who would bring to the group a fresh 
view of ongoing educational research would be important to the pro- 
gram of an institute of this type. His visit should be late enough in 
the program to allow the participants to develop the language and 



basic knowledge to insure their ability to interact with him. 

5. Statistical laboratory time should be reduced by 
about one -half. 

6. A one-half time person should be added to the 
teaching staff. 



PROGRAM REPORTS 



Many of the facts shown below are pointed out in the context of 
the evaluation. For purposes of immediate reference these items 
are repeated in summary form. 

1 . Publicity 



No public notice of the institute was made because of fac- 
tors mentioned earlier in this report. Publicity was limited to 
the letter of announcement shown in Appendix C and to oral com- 
munication. 

2. Application Summary 



a. Approximate number of inquiries from pros- 
pective trainees 124 

b. Number of completed applications received . . 52 

c. Number of first rank applications 33 

d. Applicants offered admission 35 



3. Trainee Summary 

a. Number of trainees initially accepted in the 
program 30 

Number of trainees enrolled at the beginning . 24 
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program 

b. 



Number of trainees who completed the 



Categorization of trainees 



24 



1) . Elementary or secondary school 

teachers 

2) . Local public school administrators 

7 fl 

and/or supervisors 

4. Program Director's Attendance 

a. Number of instructional days for the pro- 
gram (July 4 excluded) ^9 

b. Percent of days the director was present .... 100% 
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FINANCIAL SUMMARY 
(tentative) 



Committed 

and 





Budgeted 


Expended 


Trainee Support 






a. Stipends 

b. Dependency allowance 

c. Travel 


$18, 000. 00 
9, 000. 00 
720. 00 


$14,400.00 
6, 720. 00 
187. 19 


Total 


$27, 720. 00 


$21, 307. 19 


Direct Costs 






a. Personnel 

b. Supplies 

c. Equipment 

d. Travel 

e. Other 


$16,469. 00 
500. 00 
100. 00 
800. 00 
780. 00 


$12, 377.67 
493.21 
28. 00 
587.25 
583.00 


Total 


$18, 649. 00 


$14, 069. 13 


Indirect Costs 


$ 3,710.00 


$ 2, 814. 00 


Total 


$50, 079. 00 


$38, 190. 32 



FINANCIAL SUMMARY 
(tentative) 



1 . 



2 . 



Trainee Support 

a. Stipends 

b. Dependency allowance 

c. Travel 

Total 

Direct Costs 

a. Personnel 

b. Supplies 

c. Equipment 

d. Travel 

e. Other 

Total 

Indirect Costs 

Total 



Budgeted 


Committed 

and 

Expended 


$18, 000. 00 
9, 000. 00 
720. 00 


$14, 400. 00 
6, 720. 00 
187. 19 


$27, 720. 00 


$21, 307. 19 


$16,469. 00 
500. 00 
100. 00 
800. 00 
780. 00 


$12, 377. 67 
493. 21 
28. 00 
587. 25 
583. 00 


$18, 649. 00 


$14, 069. 13 


$ 3, 710. 00 


$ 2, 814. 00 


$50, 079. 00 


$38, 190. 32 



3. 



APPENDIX A- 1 



Statistics 



Outline of Course Content 



I. Descriptive Statistics 

A. Elementary derivations 

B. Frequency distributions - emphasis on charac- 
teristics and the introduction of grouping errors 

C. Central tendencies 

D. Variability - emphasis on development, limitations, 
and characteristics 

E. The Normal Curve - its predetermined characteris- 
tics, an outline of the mathematical development, its 
use in predictions, and the testing of hypotheses 

F. Derived scores - their purpose and use 

G. Regression and prediction - 

1. Theory of least squares 

2. Non-linear trends 

3. Concrete and theoretical illustrations 

n. Probability 

A. Finite sample spaces 

B. Combinations, approximations 

C. Conditional probability 

D. Independent events 

E. F ree schematics 

F. Binomial distribution 

G. Geometric treatments 

H. Finite Markov chains 

I. Bayes theorem 

III. Statistical Inference 

A. Sampling distributions 

B. Hypothesis testing 

C. Student's T distribution 

D. Estimation 

£. Chi Square distribution 

F. Analysis of variance, introduction only 
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APPENDIX A-2 



Measurement 



Outline of Course Content 



I. Development of measurement in education 

II. Characteristics of acceptable measurement 

A. Validity 

1. Kinds of validity 

2. Characteristics of a criterion 

3. Investigating the validity of a measure 

B. Reliability 

1. Types of reliability 

2. Meanings imparted by the type of reliability 
information 

3. Investigating the reliability of a measure 

C. Practicality 

m. Norms. . . Purposes and characteristics 

A. Age norms 

B. Grade norms 

C. Percentiles 

D. Standard scores 

E. Normalized standard scores 

IV. Constructing test items and questionnaires 

A. Taxonomy of educational objectives 

B. Types and levels of items 

C. Item writing 

D. Analysis of tests and items 

E. Characteristics important to a questionnaire 

V. School achievement measures 
A. Types of tests 
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B. Interpretation of test results 

C. Multi-test interpretation 

D. Recent trends 

VI. Measurement of intelligence 

A. Nature of intelligence. . . the construct 

B. Individual intelligence tests 

C . Group tests 

D. Omnibus tests vs. factor tests 

E. Interpretation of scores 

F. Intelligence test scores in research 

VII. Methods of personality appraisal 

A. Self report 

B. Opinion of others 

C. Direct measures 

D. Indirect measures 

VIII. School-wide testing program 

A. Selection of tests 

B. Interpreting and using test scores 

IX. Measuring teacher behavior 

A. General attempts, affective and cognitive 

B. Interaction analysis 

X. Measurement by Q-sort 

A. Characteristics of the technique 

B. Applications 

XI. Measurement by the semantic differential 

A. The technique 

B. Applications 
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APPENDIX A-3 



Research Methods 



Outline of Course Content 



The science of research methodology 

Research in education 

A. Its history 

B. Current emphasis 

C. In local schools 

Conceptualizing education as a research base 

A. Overall school program and purposes 

B. Elements of school -- a model for research 

C. Variables relating to each element of the model 

D. Identifying problems in todays schools 

The problem component of a research study 

A. Analysis of the problem 

1. Identifying variables within the problem 

2. Tracing antecedents of the problem 

B. Establishing limits for the study 

C. Significance of the problem 

D. Recognition of goals to be achieved 

E. Defining terms and relationships 

Developing hypotheses 

A. Reasons for hypotheses 

B. Types of hypotheses 

C. How do you develop an hypothesis 

D. Practice in developing hypotheses 

Selecting a research model or generating one 

A. Characteristics of a model 

B. Usefulness of the model 



VII. 



Selecting a research strategy 

A. Historical research 

B. Descriptive research 

1 . Surveys 

Z. Job B-Hs^lysis 

3. Public opinion research 

4. Community analysis 

C. Interrealtionship studies 

1. The case study 

2. Causal-comparative studies 

3. Correlational studies 

D. Developmental studies 

1. Growth studies 

2. Trend studies 

E. Experimental research 

1. Nature and control 

2. Methods of control 

3. Factors to be controlled 

4. Design with minimum control 
One-group pretest-posttest design 

5. Designs with rigorous control 

a) . Randomized control-group pretest- 

posttest design 

b) . Randomized Soloman four-group design 

c) . Randomized control group posttest only 

des ign 

d) . Factorial designs 

6. Designs with partial control 

a) . Nonrand omized control- group pretest- 

posttest design 

b) . Counterbalanced design 

c) . One-group time-series design 

d) . Control-group time-series design 



APPENDIX A -4 



Evaluation of Research 



Outline of Content 



Why evaluate research? 

Criteria for the evaluation of a research effort 

A. The problejin 

1. Clearly defined? 

2. Was a verifiable hypothesis formulated? 

3. Was the hypothesis logically developed 
from some theory? 

B. The design 

1. Suitability of the statistical design 

2. Clear statement of the population charac- 
teristics 

3. Sampling method 

4. Characteristics of the control group 

5. Random assignment of treatments 

6. Statistical inference sound 

C. The procedure 

1. Adequate description of the treatment and 

methods 

2. Controls adequate? 

3. Appropriate measurement devices 

D. The analysis 

1. Were all significant elements of the situa- 
tion analyzed? 

2. Were relationships investigated? 

3. Were . .lalysis techniques appropriate for 
the sample, measures, and population? 

E. The interpretation 

1. Were conclusions logically drawn from the 
findings? 

Generalizations adequately related to limita- 
tions of the study? 



2 . 



APPENDIX A- 5 



Research Writing 



Outline of Content 



I. Guidelines for research writing 



A. 


A format selected, organization 




B. 


Appropriate language for the topic 


and the reader 


C. 


Clarity of expression necessary 




D. 


Avoidance of extraneous material 




E. 


Grammar, punctuation, spelling 


material 


F. 


Appropriate handling of reference 


G. 


Tables, graphs and figures 




H. 


Interpretation of tables, etc. 





II. A writing experience 
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APPENDIX B 



riTTVC nTTOMTS.! A TT> TT 

RESEARCH TRAINING INSTITUTE 
SUMMER, 1966 



Your responses to these items will be used in the process of 
evaluation of the Research Training Institute with which you 
have been associated. Your candid observations as you record 
them here can be of real value in the improvement of future ef- 
forts to provide research training. Please be as specific as the 
format of this instrument will allow. Do not in any way identify 
the responses as your product. 



I. 



General Elements of the Institute: 



A. Administration 

Several administrative problems were en- 
countered in the staging of this Institute. Please 
indicate the direction and degree of influence each 
of these problems may have exercised on the out- 
comes of the Institute: 
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r 






Degree 


of Influence 






No 


Moderate Very strong 




Pr oblexXi 




T f 1 


no'nr'o 


1. 


Late notification 
of participants. 


1 


2 


3 4 5 










2. 


Less than the full 
number of quali- 
fied participants. 


1 


2 


3 4 5 










3. 


Obtaining admis- 
sion to graduate 
school. 

Shifting from 


1 


2 


3 4 5 


4. 


1 


o 


'1 A 




room to room. 
Finding space 




0 4: D 


5. 










for small group 
sessions. 


1 


2 


3 4 5 


6. 


Delays in stipend 
check . 


1 


2 


3 4 5 


7. 


Other (Specify) 






















1 


2 


3 4 5 




(State any other administrative problems which you con- 




sider significant) 









Allocation of time for various elements of the ^ram was 
a necessary part of the structure of the Institute. Pi indicate 

your opinion of the amount of time devoted to each of th._ content 
areas and activities. If you feel the time allotted was too short 
place a check in the column ’’Increased” and a check in the column 
headed by the percents: 25%, 50%, 75%, 100%. If you feel the 

time allotted was too long check the column ’Decreased” and in- 
dicate the percent by which it should be decreased. If you feel 
the time allottment was ’’appropriate” check that column and 
leave the percent columns blank. 
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ERIC 



other (Specify) 



B. 



Content of the Program 

The content of the program was guided by four- 
teen objectives stated in behavioral terms. Refer- 
ring to the attached page for statements of these ob- 

• j* i_ - 1 ^ 4- U in f\x;o 

J0C tlV0S IXictKt; y ^^U .1 c; V a,xu.<a,i,j.v^ax ^ 

actions: 

1. On the line to the left of the page place a check 
on the scale indicating the degree to which the 
objective has been reached. 

2. On the scale to the right of the page place be- 
neath each content area a numeral selected 
from the following list: 

1. contributed very little to the objective 

2. significant but minor contribution to 



3. 

4. 

5. 



the objective 

moderate contribution to the objective 



a strong contribution to the objective 



was the major contribution to the ob- 



jective 
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Statistics 


Measurement 


Research Design 


Research Methods 


Research Writing 


Statistics - Lab 


Field Trips 


Consultants 


Research Evaluation 


Unachieved 




Moderate 




Fully 

Achieved 


Objective 


1 


2 


3 


4 


5 


1 1 




















1 


2 


3 


4 


5 


2 n 




















1 


2 


3 


4 


5 


3 




















i 


2 


3 


4 


5 


4 




















1 


2 


3 


4 


5 


5 




















1 


2 


3 


4 


5 


6 




















1 


2 


3 


4 


5 


7 




















1 


2 


3 


4 


5 


8 




















1 


2 


3 


4 


5 


9 




















1 


2 


3 


4 


5 


10 




















1 


2 


3 


4 


5 


11 




















1 


2 


3 


4 


5 


12 




















1 


2 


3 


4 


5 


13 




















1 


2 


3 


4 


5 


14 





















Instructional Methods 



Please indicate with a check your reaction to each of the 
following items; 

1. The use of lecture as a teaching method was 

too frequently 

about right 

too infrequently 
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2. Class discussion was used as a teaching technique 

too frequently 

about right 

too infrequently 



3. Use of teaching materials such as films, de- 
vices, duplicated materials and machines was 

too extensive 

about right 

insufficient 



4. Instructors uses of practical applications, 
illustrations, examples, analogies, and 
comparisons were too extensive 

about right 

insufficient 



5. Opportunities to ask questions and to re- 
ceive individual help were abundant 

adequate 

infrequent 



Personal Involvement 



Indicate your degree of personal involvement and growth 
by placing check marks in the appropriate spaces. 

1. Degree of personal involvement in the activities of the In- 
stitute: 

more than I have been involved in a course in the past 
about like my typical degree of involvement in courses 
less than my typical degree of involvement in courses 

2. My personal involvement would have been increased if 

a number of other concerns had not interfered 
~ the material had been more directly suited to my needs 

the material had been more interesting presented 
the material had been less difficult for me 
the material had been more challenging 
other (specify)_ 



3. Please indicate your personal development during the In- 
stitute by placing a circle on the scales to represent your 
position at the beginning and an "x” to represent your pre- 

A A V ^ Jk VA X^AA • 

A, Knowledge of subject matter of the Institute 

Low Below Average Average Above Average High 
1 2 3 4 5 



B. Attitude toward research in education 

Strong Strong 

Negative Accepting Positive 

1 2 3 4 5 



C. Ability to conceptualize problems in education 

Low Below Average Average Above Average High 
1 2 3 4 5 



D. Ability to evaluate research of others 

Low Below Average Average Above Average High 
1 2 3 4 5 



E. Ability to design and carry out research in education 



Low Below Average 
1 2 


Average 

3 


Above Average 
4 


High 

5 


F. Ability to communicate 


research matters to others 


Low Below Average 


Average 


Above Average 


High 


1 2 


3 


4 


5 



What is your chief criticism of the Research Training Institute? 
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What, in your estimation, is the most favorable element of 
the institute? 



What recommendation would you make for changes in a future 
Institute designed for the same objectives as this one? 
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tr tr* "NT TT V 

XV XJ X'<l X U XV X 



Lexington, Kentucky 40506 



College of Education 



April 20, 1966 



Dear Colleague 

Enclosed are copies of the notice of a Research Training Insti- 
tute to be offered during the regular summer session at the 
University of Kentucky. This certainly is late to be announcing 
such an opportunity, but an unfortunate delay at the federal level 
has caused the problem. 

This Institute should prove to be highly valuable to persons who 
have research responsibilities or who might assume them later. 
We request your help in getting this information to the appro- 
priate persons through direct contact from you or posting in the 
right places. 

Your cooperation is appreciated. 



Sincerely 



Ivan L. Russell, Director 
Research Training Institute 



mw 



Enclosure 



APPENDIX C-2 



IMPOR TANT 

NOTICE! NOTICE! NOTICE! NOTICE! 



The 

College of Education 
University of Kentucky 

will offer a Research Training Institute 

for 

Persons who direct research and development programs in 
local school systems or state departments of public instruction 

Dates: June 13, 1966, to August 5, 1966 (8 weeks) 



General Information 

This is a program designed as initial training of persons 
designated as responsible for the development of Title I proposals, 
research projects, and developmental activities for local schools. 
Content of the Institute will include: research process, evaluation 

of research, research writing, statistical methods, measurement, 
and research design. The Institute is supported by the United 
Str.tes Office of Education under Title IV of the Elementary and 
Secondary School Act of 1965. The deadline for applications is 
May 12, 1966. 

Stipends of $75 per week and $15 per week for each depen- 
dent will be paid to each participant. Travel one-way at the rate 
of 8 cents per mile to the Institute from the participant's home 
will be paid. 

Graduate credit of nine-semester hour's will be granted 
to those participants who are admitted to the Graduate School at 
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For further information and/or application blanks write 
immediately to: 

Dr. Ivan L. Russell, College of Education 
University of Kentucky, Lexington, Kentucky 40506 



APPENDIX C-3 



RESEARCH TRAINING INSTITUTE 

"University of Kentucky 
Lexington, Kentucky 



The Research Training Institute to be conducted Junj 13, 
1966, to August 5, 1966, at the University of Kentucky is directed 
to the needs of local school system researchers. While the con- 
tent will be concentrated upon statistics, measurement o.nd re- 
search design, a very practical approach to these topics will be 
evident. Lecture, discussion, laboratory, field trips, and con- 
sultants will be involved in the eight- week institute. 



Requirements for Participation 

1. A baccalaureate degree and two years of teaching 
or administrative experience. 

2. Responsibility in the research program of a local 
school system. 

3. If graduate credit is sought, it is necessary to be 
admitted to the Graduate School of the University 
which requires: 

(a) Graduate from an accredited college 

(b) An overall 2. 5 grade point average on a 4. 0 
point scale. 

Final selection of candidates will be made by the Director 
of the Institute. 
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Notification should be expected by May 20, 1966. 



Courses for which credit naay be sought are (nine hours 
from among): 



Education 522 
Education 656 

Education 657 
Education 658 



Tests and Measurements 
Methodology of Educational 
Research 

Educational Statistics 
Problems in Educational 
Psychology 



3 hours 

3 hours 
3 hours 

3 hour’s 



Participants must make the " own housing arrangements. 
Stipends will be paid at the rate of $75 per week and $15 per week 
for each dependent who accompanies the participant to Lexington. 
Travel will be paid at the rat*^ of 8 cents per mile for one way 
from the place of residence to Lexington, Kentucky. 

Complete the enclosed forms and return them immediately 



Dr. Ivan L. Russell, Director 
Research Training Institute 
111 Dickey Hall 
College of Education 
University of Kentucky 
Lexington, Kentucky 40506 
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APPENDIX G-4 



RESEARCH TRAINING INSTITUTE 

College of Education 
University of Kentucky 
Lexington, Kentucky 40506 



APPLICATION 



Name 



Address 



Employer 



Sex: M F Age 
T elephone 



Telephone 



List Dependents 



Name 



Education 



Institution 



Degree 



Field 



Experience (last two positions only) 



Employer 



Duties 



Please place a check mark before each of the following courses 
if you have taken such a course in your graduate training. 

Tests and Mea,surements 

^Introductory Statistics 

Research Methods in Education 

List other research training courses you have taken. 



Will you register for course credit if you are. admitted to the 
Institute? ^Yes ^No 

Briefly describe your interest in a research training institute. 



Date 



Signa,ture 



